The invention relates to the field of computer networking. In particular, the invention relates to the field of nodes and switches for Fibre Channel storage-area networks. With more particularity, the invention relates to data structures for use by nodes in dynamically tracking network resources and configuration so as to properly encode and route packets over the network.
Most modern computer networks, including switched and arbitrated-loop fibre-channel networks, are packet oriented. In these networks, data transmitted between machines is divided into chunks of size no greater than a predetermined maximum. Each chunk is typically packaged with a header and a trailer into a packet for transmission. In Fibre-Channel networks, packets are known as Frames.
A Fibre-Channel network having at least one switch is a switched Fibre-Channel fabric. A Fibre-Channel switch is a routing device generally capable of receiving frames, storing them, decoding destination information from headers, and forwarding them to their destination or another switch further along a path toward their destination. A network interface for connection of a machine to a Fibre Channel fabric is known as an N_port, and a machine attached to a Fibre-Channel network is known as a node. Nodes may be computers, or may be storage devices such as RAID systems. An NL_port is an N_port that supports additional arbitration required so that it may be connected either to a Fibre Channel Fabric or to a Fibre Channel Arbitrated Loop, and an L_port is a network interface for connecting a node to a Fibre Channel Arbitrated Loop.
A device including an N_port, L_port, or an NL_port together with hardware for high-speed connection to a machine is a fibre channel host bus adapter (physical HBA). For example, a physical HBA may comprise a printed circuit card having one or more NL_ports communicating through a PCI bus interface to an edge connector for connection to a PCI bus of a machine. A physical HBA may, but need not, also incorporate a processor for controlling its ports and its connection to the machine.
A Fibre Channel Switched Fabric may connect one or more Fibre Channel Arbitrated Loops.
In a switched fibre channel fabric, there may be more than one possible path, or sequence of links, loops, switches, routers, etc. that may be traversed by a frame, between two nodes. Multiple paths may be intentional, providing extra capacity or redundancy to protect against switch, node, or line failures, or may be unintentional consequences of network topology.
Multiple paths between two nodes may also be provided through multiple fibre channel arbitrated loops. For example, an initiator node may have two NL_ports, one connected to each of two fibre channel arbitrated loops. If each of these loops connects to an NL_port of a target node, then multiple paths from the initiator to the target node exist. This could provide redundancy should a failure occur on one of the arbitrated loops.
Fibre Channel storage-area-network (SAN) nodes and switches, especially network nodes having multiple ports, must keep track of a variety of information about the network and resources available over the network. This information is used by each node to format and properly route frames onto and over the network.
Typically, a program references files in storage by passing a command block to an operating system. At this level, the command block may reference files by name and device name, or by handle for files already opened. The operating system passes block I/O requests derived from the command block to an appropriate driver. The command block includes a command field, a file name field, and a drive name or number field in addition to other fields. The block I/O requests derived from the command block typically include fields for device identification, command, and a count of logical blocks to be operated upon, and may include pointers to data.
The device driver typically converts the block I/O requests into a sequence of one or more device level commands. Once the driver attempts to execute each command, it returns to the operating system a response having status information, and other information possibly including data read from storage. Information from the response may then be transferred to the program.
When a command block references storage accessible over a Fibre Channel network, the driver must encapsulate the device level commands into one or more command frames, and for write operations one or more data frames. The driver may use the network information to determine header information and routing for the one or more fibre channel network frames, or packets, that implement the command.
Typically, the operating system passes block I/O requests to the device driver with a device tag identifying the specific device intended to perform the desired operation. This tag may comprise a referenced device name, handle, or SCSI nexus, where a SCSI nexus includes bus identity, target device number, and logical unit number.
In particular, the driver must translate device tags, into a multilevel address as required to reach the indicated device. In a Fibre Channel context, that multilevel address field must include valid Destination Identification (D_ID) fields and routing information field for each frame. A command frame may specify a specific logical unit. There may also be additional destination address fields such as those in association headers; association headers permit addressing multiple devices or processes through a single fibre channel port.
The driver, especially a multiport driver servicing multiple ports, must therefore determine an appropriate destination and routing for each frame required to implement a command, and transmit those frame over a port appropriate for that routing. It is desirable that these translations and assignments be done quickly and accurately using network topology information maintained in a local topology database.
The command and data frames transmitted by a node in implementing an I/O command, together with any frames transmitted by another node in response to those command and data frames, is known as an exchange.
As nodes, switches, and links are added to or removed from the network, any local topology database must be updated to reflect valid devices on the network, and valid paths through the network to those devices. Nodes also may determine one or more paths of the valid paths to a given device to be an xe2x80x9cactivexe2x80x9d path. An active path is a path that may be used for exchanges.
The Fibre Channel specifications define Class 1 and Class 4 services to be virtual-circuit, or connection, based services between pairs of nodes. Packets of a given sequence in these services must arrive in-order with respect to other packets of the same sequence. The specifications presume that frames transiting between nodes of each pair follow a virtual circuit between the nodesxe2x80x94all following the same path through the network from node to node and arriving in-order.
Links, loops, and switches of a network may fail. Fibre channel networks may provide more than one path between a pair of nodes. Multiple, redundant, paths provide redundancy to allow continued communications between a pair of nodes should one or sometimes more, but not all, links, loops, or switches, and therefore paths through the network, fail.
Some existing fibre-channel systems can recognize failure of a path, switching traffic between a pair of nodes to an alternate path if one exists. This is known as failover of a path. Unfortunately, switching Class 1 and Class 4 fibre channel connections, and associated network traffic, to alternate paths is known to cause considerable delay to ongoing exchangesxe2x80x94on the order of seconds. These delays are caused in part by the need to ensure that Class 1 and Class 4 frames arrive at their destination in-order; time is allowed for flushing in-transit frames before frames are transmitted on the alternate path.
Further delay may occur if redundant controllers having access to the target device must transfer ownership of a logical unit as part of a path failover.
While a link or switch of a fibre channel network may fail, it may also be repaired. It is desirable that paths associated with repaired links, loops, or switches be dynamically returned to use to restore network redundancy and capacity.
Loops, links or switches may develop intermittent problems, where they may alternate between repaired and failed conditions. Considerable delay of network traffic can be encountered if paths through an intermittent device are used, since the delays associated with switching traffic to repaired paths, thence back to alternate paths upon the next failure may be cumulative. Paths may also be reported as repaired but actually have higher error rates. High error rates cause delay as frames are retransmitted as errors are detected; high error rates may also cause a path to be considered to have failed. It is therefore desirable to control use of repaired paths such that extended delay does not occur.
It is known that many nodes, including RAID storage subsystems, have the ability to queue multiple commands in a command queue for subsequent execution. For example, a RAID system may queue several read or write commands, received from one or more machines. Once queued, these commands are executed from the queue to or from cache, or to or from disk, in an order depending on availability of data in cache, disk availability and disk rotation. With proper interlocks, execution may often be in an order different from that in which the commands were received. Commands placed in a command queue of a node by an initiator node and not yet completed are pending commands. For this purpose, commands that are aborted, including those aborted for timeouts, are considered completed.
Commands that may be queued in these devices may include commands from multiple processes, or threads, running on a single node having one or more processors. For example, a transaction-processing system may have several processes running, each process requiring access to a different record of a database on a RAID system, all requesting access to the database at about the same time. Each process may create read, write, lock, or unlock commands for the database. Commands may also be queued from multiple nodes, where each node is running one or more processes or threads that require access to the device. Queuing and execution of each of these commands requires an exchange of frames between the machine and the device.
The maximum number, or queue depth, of commands that may be pending in any one device at any one time is finite and characteristic of the device. Each node may, but need not, know the queue depth of the device nodes it is attempting to use, and may throttle its commands appropriately. Typically, each node originating commands and transmitting them to the command queue of a device is unaware of the commands originated by other nodes and assumes it has the full queue depth of the device available to it.
If a node assumes that the full queue depth of a device is available to it, the queue depth of the device may be exceeded by commands from multiple nodes. When this happens, the excess commands are refused by the device and must be retransmitted later; the need for delay and retransmission degrades overall system performance. Conversely, if a node assumes that a device has only a small queue depth available to it, commands may be throttled unduly, such that overall system performance is impaired. It is desirable to adjust the maximum number of commands from each node transmitted to a device command queue in light of the command traffic to that device from other nodes of the network.
Manually tuned allocations of queue depth available to individual initiator nodes of a storage area network is possible, but is laborious and error prone. Further, it is known that the load on a system may vary from time to time throughout a day; it is desirable to change queue depth allocations as load changes for optimum performance.
A driver for one or more Fibre Channel N_ports, L_ports, or NL_ports maintains a local network topology database. This database includes linked lists of host bus adapter (HBA) port records, target node records, and device records.
HBA port records correspond to individual physical N_ports, L_ports or NL_ports and are linked to node records through linked lists of node links. Similarly, node records are linked to device records through linked lists of device links, and devices are linked to node links through linked lists of path links. Additional HBA port records may be added as ports are discovered or as hot-pluggable ports are added to the system.
Each node record has information pertinent to a particular port of a target node of the fibre channel network. A target node may have more than one port, and may therefore have more than one node record. Similarly, each device record has information pertinent to a particular device that can be accessed from one or more nodes. Each path link has information, including status information, pertinent to a particular path through a fibre channel fabric or arbitrated loop between a physical HBA port and a physical node or device of the network.
In order to reduce command queue overflow errors, a command queue depth variable is maintained for each node record. Initially, this is set to match a maximum queue depth determined from the type of node represented by the record. This queue depth variable is adjusted downwardly when queue overflow errors occur. Queue depth is also adjusted downwardly when other initiators are first detected on the fibre channel network, so as to allow those initiators to access the node. The queue depth is adjusted upwardly when a predetermined period elapses without queue overflow errors.
Nodes that have successfully accessed a target node without queue overflow errors automatically try successively higher values of queue depth until they find a depth for optimum performance. Further, queue depth is automatically reallocated to active nodes as other nodes become idle.
In order to limit dispatch of frames over intermittent or problem paths, with attendant errors and retries, freshly repaired and problem links and their associated paths are given probationary status. Frames are preferentially dispatched over non-probationary paths if any exist, if no non-probationary paths exist a probationary path may, in desperation, be returned to normal status. If no probationary paths exist, failed links are tested to determine if they have been repaired; a repaired path may then be activated bypassing the probationary state.
Probationary paths that function without error for a preset period of time are returned to normal status, where they may once again be subjected to normal use. Probationary paths that record excessive error counts are returned to failed status, unless these errors occur within a few seconds of another device logging into a fibre channel fabric; this prevents marking paths failed solely because of transient errors induced by nodes bringing their laser diodes online.
The foregoing and other features, utilities and advantages of the invention will be apparent from the following more particular description of a preferred embodiment of the invention as illustrated in the accompanying drawings.